Self-Driving Car Engineer Nanodegree¶
Deep Learning¶
Project: Build a Traffic Sign Recognition Classifier¶
In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.
In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.
Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.
Step 0: Load The Data¶
# Load pickled data
import pickle
# TODO: Fill this in based on where you saved the training and testing data
training_file = '../traffic-signs-data/train.p'
testing_file = '../traffic-signs-data/test.p'
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']
Step 1: Dataset Summary & Exploration¶
The pickled data is a dictionary with 4 key/value pairs:
'features'is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).'labels'is a 2D array containing the label/class id of the traffic sign. The filesignnames.csvcontains id -> name mappings for each id.'sizes'is a list containing tuples, (width, height) representing the the original width and height the image.'coords'is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES
Complete the basic data summary below.
### Replace each question mark with the appropriate value.
import numpy as np
# TODO: Number of training examples
n_train = X_train.shape[0]
# TODO: Number of testing examples.
n_test = X_test.shape[0]
# TODO: What's the shape of an traffic sign image?
image_shape = X_train.shape[1:]
# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(np.unique(y_train))
print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.
NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.
### Data exploration visualization goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
import math
# Visualizations will be shown in the notebook.
%matplotlib inline
# distribution of classes in training and sets
fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(16, 4))
ax0.hist(y_train, n_classes, histtype='bar', facecolor='g', alpha=0.75, rwidth=0.5)
ax0.set_title('Label distribution in training set')
ax0.set_xlabel('Label')
ax0.set_ylabel('Frequency')
ax1.hist(y_test, n_classes, histtype='bar', facecolor='g', alpha=0.75, rwidth=0.5)
ax1.set_title('Label distribution in test set')
ax1.set_xlabel('Label')
ax1.set_ylabel('Frequency')
plt.show()
# read label codes from csv
import pandas as pd
sign_names_df = pd.read_csv('signnames.csv')
classIdToSignNameDict = sign_names_df.set_index('ClassId').to_dict()['SignName']
def show_images(images, titles, cols):
assert len(images) == len(titles)
num_images = len(images)
rows = math.ceil(num_images/cols)
fig, axes = plt.subplots(rows, cols)
fig.set_figheight(rows * 5)
fig.set_figwidth(cols * 5)
idx = 0
for i, ax in enumerate(axes.flat):
if idx < num_images:
ax.imshow(images[idx])
ax.set_xlabel(titles[idx])
ax.set_xticks([])
ax.set_yticks([])
idx = idx + 1
plt.show()
print("Grab one image per class in training set...")
sample_examples = {}
for idx in range(n_train):
if y_train[idx] not in sample_examples:
sample_examples[y_train[idx]] = X_train[idx]
images = []
titles = []
for key in sorted(sample_examples):
titles.append(key)
images.append(sample_examples[key])
print("Show sample images and classes")
show_images(images, titles, 4)
Observations:
- There is a huge variance in the amount of data available per class. For example class es 1,2,4 have > 2000 training samples, but classes 0,5,20 have < 500 training samples. Because of this, I expect the model to have better accuracy for classes with more training data, and lower accuracy for classes with less training data.
- Many images in the training and test set have poor lighting, and its hard for even human eye to recognize these sometimes.
Step 2: Design and Test a Model Architecture¶
Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.
There are various aspects to consider when thinking about this problem:
- Neural network architecture
- Play around preprocessing techniques (normalization, rgb to grayscale, etc)
- Number of examples per label (some have more than others).
- Generate fake data.
Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.
NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!
Implementation¶
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.
from sklearn.preprocessing import LabelBinarizer
from sklearn.utils import shuffle
### Generate data additional data (OPTIONAL!)
### dummy for now
generated_data_X = X_train[0:1]
generated_data_y = y_train[0:1]
### Set up train, test and validation data
from sklearn.model_selection import train_test_split
### combine train data from dataset to generated. same with test
all_X_train = np.concatenate((X_train, generated_data_X))
all_X_test = X_test
all_y_train = np.concatenate((y_train, generated_data_y))
all_y_test = y_test
### and split the data into training/validation/testing sets here.
### split train into train (90%) and validation(10%)
### This will leave us in all with
### train set at (0.9*0.75) 67.5%,
### validation set at (0.1*0.75) 7.5%
### and test set at 25%
train_features_orig, valid_features_orig, train_labels, valid_labels = train_test_split(
all_X_train,
all_y_train,
test_size=0.1,
random_state=832289)
test_features_orig = all_X_test
test_labels = all_y_test
print("Training examples shape =", train_features_orig.shape)
print("Validation examples shape =", valid_features_orig.shape)
print("Test examples shape =", test_features_orig.shape)
print("Test labels shape =", test_labels.shape)
### Preprocess the data here.
import cv2
# make mean 0 and values range [-0.5, 0.5]
def make_zero_mean(imgs):
return (imgs - 128.) / 255.
# there are a lot of images with low exposure.
# normalize Y value in YUV format, and retain same values for U and V
def yuv_normalize(yuv_img):
y_vals = yuv_img[:,:,0].flatten()
y_min = y_vals.min()
y_max = y_vals.max()
normalized = (yuv_img - np.array([y_min, 0, 0]))/np.array([y_max - y_min, 1, 1])
return normalized * [255, 1, 1]
def convert_to_yuv(image):
return cv2.cvtColor(image, cv2.COLOR_BGR2YUV)
def print_stats(vec):
return "min: %d, max: %d, mean: %.2f"%(vec.min(), vec.max(), vec.mean())
def print_stats_yuv(yuv_img):
y_vals = yuv_img[:,:,0].flatten()
u_vals = yuv_img[:,:,1].flatten()
v_vals = yuv_img[:,:,2].flatten()
print(print_stats(y_vals))
print(print_stats(u_vals))
print(print_stats(v_vals))
# preprocess images
def preprocess_images(imgs):
yuv_imgs = imgs.copy()
for idx in range(imgs.shape[0]):
yuv_imgs[idx] = convert_to_yuv(imgs[idx])
yuv_imgs_normalized = yuv_imgs.copy()
for idx in range(yuv_imgs.shape[0]):
yuv_imgs_normalized[idx] = yuv_normalize(yuv_imgs[idx])
return make_zero_mean(yuv_imgs_normalized)
train_features = preprocess_images(train_features_orig)
test_features = preprocess_images(test_features_orig)
valid_features = preprocess_images(valid_features_orig)
# one-hot encode labels will be done in tensor flow code
Question 1¶
Describe how you preprocessed the data. Why did you choose that technique?
Answer:
All images are nicely preprocessed to 32x32x3 size for us.
Since color plays an important role in identifying traffic sign, I am retaining all the RGB values instead of coverting to gray scale image. Since there are many pictures with low exposure, I am converting them to YUV scale.
I am normalizing Y using min-max normalization between 0 and 255. I am retaining the values of U anv V.
Finaly, I normalized feature values of Y, U and Vto have a mean of 0 and very small sigma. This well conditioned data will help gradient descent reach minima quickly and reliably. Since these values all range from [0, 255], I applied a simple normalization of :
val = (val - 128 ) / 255
This will make the mean 0, and values range [-0.5, 0.5]. I converted label to one-hot encoding.
Question 2¶
Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?
Answer: The original test data set contains approximately 25% of the total data, and training dataset has 75%.
I used 10% of training data set as validation dataset. The other 90% is used as training dataset. I retained all the testing data provided as test dataset.
train set = 0.9 * 0.75 = 67.5% of all data
validation set = 0.1 * 0.75 = 7.5% of all data
test set = 25% of all data
### Define your architecture here.
### Feel free to use as many code cells as needed.
from tensorflow.contrib.layers import flatten
def PierreYannLikeArchitecture(x, mu, sigma, nclasses, keep_prob):
# Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 3, 6), mean = mu, stddev = sigma))
conv1_b = tf.Variable(tf.zeros(6))
conv1 = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b
# Activation.
conv1 = tf.nn.relu(conv1)
# Pooling. Input = 28x28x6, Output = 14x14x6.
conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')
# Flatten. Input = 14x14x6. Output = 1176
flat_c1 = flatten(conv1)
flat_c1 = tf.nn.dropout(flat_c1, keep_prob)
# Convolutional. Input = 14x14x6 Output = 10x10x16.
conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
conv2_b = tf.Variable(tf.zeros(16))
conv2 = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
# Activation.
conv2 = tf.nn.relu(conv2)
# Pooling. Input = 10x10x16. Output = 5x5x16.
conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')
# Flatten. Input = 5x5x16. Output = 400.
flat_c2 = flatten(conv2)
flat_c2 = tf.nn.dropout(flat_c2, keep_prob)
# Convolutional. Input = 5x5x16 Output = 3x3x30.
conv3_W = tf.Variable(tf.truncated_normal(shape=(3, 3, 16, 30), mean = mu, stddev = sigma))
conv3_b = tf.Variable(tf.zeros(30))
conv3 = tf.nn.conv2d(conv2, conv3_W, strides=[1, 1, 1, 1], padding='VALID') + conv3_b
# Activation.
conv3 = tf.nn.relu(conv3)
# Flatten. Input = 3x3x30. Output = 270.
flat_c3 = flatten(conv3)
flat_c3 = tf.nn.dropout(flat_c3, keep_prob)
# Join flat layers from flat_c1, flat_c2 and flat_c3. Inputs 1176, 400, 270. Output = 1846.
flat_combined = tf.concat(1, [flat_c1, flat_c2, flat_c3])
# Layer 3: Fully Connected. Input = 1846. Output = 200.
fc1_W = tf.Variable(tf.truncated_normal(shape=(1846, 200), mean = mu, stddev = sigma))
fc1_b = tf.Variable(tf.zeros(200))
fc1 = tf.matmul(flat_combined, fc1_W) + fc1_b
# Activation.
fc1 = tf.nn.relu(fc1)
# Dropout layer
fc1 = tf.nn.dropout(fc1, keep_prob)
# Layer 5: Fully Connected. Input = 200. Output = nclasses.
fc2_W = tf.Variable(tf.truncated_normal(shape=(200, nclasses), mean = mu, stddev = sigma))
fc2_b = tf.Variable(tf.zeros(nclasses))
fc2 = tf.matmul(fc1, fc2_W) + fc2_b
# Activation.
logits = tf.nn.relu(fc2)
return logits
Question 3¶
What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.
Answer:
I am using a conv net architecture similar to the one described in Pierre Sermanet and Yann LeCun's paper referrenced above. Each input image is a 32x32x3 image.
I apply a 5x5 convolution on it with 6 feautres, which gives a 28x28x6 output. On top of this, I subsample by max pooling, which gives me 14x14x6 output. Lets call this layer 1.
Another 5x5 convolution on output of layer1 with 16 out features gets me to a 10x10x16 output. Applying maxpool subsampling down converts this to 5x5x16 output. Lets call this layer 2.
Another 3x3 convolution on output of layer 2 with 30 output features gets me to a 3x3x30 output. Lets call this layer 3.
I flatten: layer 1 output to get flat vector of size 14x14x6 = 1176 layer 2 output to get flat vector of size 5x5x16 = 400 layer 3 output to get flat vector of size 3x3x30 = 270 I add dropout of 0.5 for each one of the flat layers above and combine the three flat vectors to form a vector of size 1176 + 400 + 270 = 1846
After this there are 2 fully connected layers. First fully connected layer connects 1846 features to 200. And I added a dropout of 0.5 there. Finally there is the last fully connected layer that connects 200 features to 43 features, which is the number of classes. This layer can be thought of as classifier layer.
### Train your model here.
### Feel free to use as many code cells as needed.
import tensorflow as tf
from sklearn.utils import shuffle
### Hyperparameters - START
EPOCHS = 200
BATCH_SIZE = 256
# params for initial weights
MU = 0
SIGMA = 0.1
LEARNING_RATE = 0.08
### Hyperparameters - END
x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, n_classes)
keep_prob = tf.placeholder(tf.float32)
logits = PierreYannLikeArchitecture(x, mu=MU, sigma=SIGMA, nclasses=n_classes, keep_prob=keep_prob)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdagradOptimizer(learning_rate=LEARNING_RATE)
training_operation = optimizer.minimize(loss_operation)
predictions = tf.argmax(logits, 1)
truth_labels = tf.argmax(one_hot_y, 1)
correct_prediction = tf.equal(predictions, truth_labels)
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
softmax_probabilities = tf.nn.softmax(logits)
def evaluate(X_data, y_data, batch_size):
num_examples = len(X_data)
total_accuracy = 0
sess = tf.get_default_session()
for offset in range(0, num_examples, batch_size):
batch_x, batch_y = X_data[offset:offset+batch_size], y_data[offset:offset+batch_size]
accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_prob:1})
total_accuracy += (accuracy * len(batch_x))
return total_accuracy / num_examples
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
num_examples = len(train_features)
print("Training...")
for i in range(EPOCHS):
X_train, y_train = shuffle(train_features, train_labels)
for offset in range(0, num_examples, BATCH_SIZE):
end = offset + BATCH_SIZE
batch_x, batch_y = X_train[offset:end], y_train[offset:end]
sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_prob:0.5})
validation_accuracy = evaluate(valid_features, valid_labels, BATCH_SIZE)
if (i%10 == 0):
print("EPOCH %d, Validation Accuracy = %.3f" %(i, validation_accuracy))
try:
saver
except NameError:
saver = tf.train.Saver()
model_path = saver.save(sess, 'model')
print("Model saved at path: ", model_path)
### test accuracy
with tf.Session() as sess:
# Restore variables from disk.
loader = tf.train.import_meta_graph('model.meta')
loader.restore(sess, tf.train.latest_checkpoint('./'))
test_accuracy = evaluate(test_features, test_labels, BATCH_SIZE)
print("Test Accuracy = {:.3f}".format(test_accuracy))
The set up mentioned above gives me validation accuracy of 99.7% on 200 epochs.
And test accuracy of 97%.
### print a few images and see performance on test set
nine_test_features = test_features[0:9]
nine_test_labels = test_labels[:9]
original_test_images = test_features_orig[0:9]
with tf.Session() as sess:
loader = tf.train.import_meta_graph('model.meta')
loader.restore(sess, tf.train.latest_checkpoint('./'))
preds = predictions.eval(feed_dict={x: nine_test_features, y: nine_test_labels, keep_prob:1.0})
titles = ["Truth: %d, Pred: %d"%(nine_test_labels[idx], preds[idx]) for idx in range(len(nine_test_features))]
show_images(original_test_images, titles, 3)
### confusion matrix
from sklearn.metrics import confusion_matrix
def normalize_by_row(arr):
sum_along_rows = np.sum(arr, axis=1)
sum_along_rows = sum_along_rows[:, np.newaxis]
return arr/sum_along_rows
with tf.Session() as sess:
# Restore variables from disk.
loader = tf.train.import_meta_graph('model.meta')
loader.restore(sess, tf.train.latest_checkpoint('./'))
preds = predictions.eval(feed_dict={x: test_features, y: test_labels, keep_prob:1.0})
conf_matrix = confusion_matrix(test_labels, preds)
conf_matrix = normalize_by_row(conf_matrix)
plt.matshow(conf_matrix)
plt.colorbar()
plt.xlabel('Predicted')
plt.ylabel('True')
print('Classes with low accuracy')
print('Class - Accuracy')
for i in range(n_classes):
acc = conf_matrix[i][i]
if acc < 0.75:
print("%d - %f"%(i, acc))
Question 4¶
How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)
Answer:
1) Optimizer : I am using adam optimizer since it moving averages of the parameters (momentum). Adam optimizer keeps track of exponentially decaying past momentums and as a result reduces learning rate at each iteration. I explored reasonable learning rates of 0.001, 0.002, 0.005 and 0.08. Adam optimizer will decay this rate over each iteration. I settled with a rate of 0.08 which helped me converge faster.
2) I trained on EC2 machine, and was able to use batch sizes of 64, 128, and 256 without significantly slowing down the system. I settled with batch size of 256.
3) With rate of 0.008, validation error reaches quickly to 99% around 30 iterations. I continued this for 200 epochs which gave me validation error of 99.7%.
4) For initial weights I am using a mean of 0 with small sigma 0.1 since its recommended to use small initial weights.
5) I am using a 50% dropout for training set, and no dropout for validation and test sets.
Question 5¶
What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.
Answer:
Trial and error: I started with LeNet architecture described in the class, which gave me a validation accuracy of around 94%. Increasing number of features at second convolution improved validation accuracy slightly. Adding 2 more fully connected layers gave me validation accuracy of 99%, which I was happy with. But test accuracy was around 90%. This is an indication that my model was overfitting training set. For regularilzation, I added a drop out layer, and initially this dropped my validation accuracy to 97%. After increasing number of epochs and size of fully connected layers, I was able to get 99% validation accuracy, and 94% test accuracy.
Pierre Sermanet and Yann LeCun's architecture: I was not content with 94% test accuracy. After reading Pierre Sermanet and Yann LeCun's paper, I liked the idea of using feature outputs from earlier layers (layer1 and 2) along with the last layer(layer 3) before passing them along to fully connected layers.
I started using 2 convolution layers and flattened both layer 1 and layer 2 outputs and send to a fully connected layer. This initial implementation gave me a test error of 94.2% without much effort. I improved this by adding a third covolution layer, and using 2 fully connected layers(one with drop out) instead of one. After playing with learning rate and number of epochs, I was able to achieve test accuracy of 97%.
By using outputs from all the convolution layers means we are building more and more complex features, and at the same time using simpler and complex features together in fully connected layers. Convolution layer 1, 2 and 3 outputs will (in my opinion) be able to learn the shape of the plate of the road sign, colors etc, but learning the shapes like children crossing is challenging. In addition to that road signs like deer crossing, pedestrain crossing, road work etc... look very similar at the resolution we have for 32x32 image, especially in poor lighting conditions. Passing layer 1 and 2 output features along to fully connected layers, IMO, helps algorithm learn these complicated signs better.
Step 3: Test a Model on New Images¶
Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.
You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.
Implementation¶
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
from os import listdir
import matplotlib.image as mpimg
import cv2
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline
import numpy as np
path_to_directory = 'new_images'
external_images = []
external_labels = []
external_images_resized = []
for f in listdir(path_to_directory):
label = int(f.split('_')[0])
file_path = path_to_directory + "/" + f
image = cv2.imread(file_path,cv2.IMREAD_UNCHANGED)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
external_images.append(image)
external_labels.append(label)
resized_image = cv2.resize(image, (32, 32), interpolation = cv2.INTER_CUBIC)
external_images_resized.append(resized_image)
external_images_resized = np.asarray(external_images_resized)
external_labels = np.asarray(external_labels)
show_images(external_images, external_labels, 3)
Question 6¶
Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.
Answer:
Our training dataset contains road signs which look like full frontal images, and the sign occupies most of the space of the image. In the sample images I picked:
1) road signs are not exactly in the middle of the image
2) signs dont exactly occupy most of the space in the image
3) the signs are in various different perspectives and angles in the sample images.
Although the sample images are well lit, the reasons mentioned above make classification difficult.
### Run the predictions here.
### Feel free to use as many code cells as needed.
import tensorflow as tf
external_X = preprocess_images(external_images_resized)
external_y = external_labels
with tf.Session() as sess:
# Restore variables from disk.
loader = tf.train.import_meta_graph('model.meta')
loader.restore(sess, tf.train.latest_checkpoint('./'))
dict={x: external_X, y: external_y, keep_prob:1.0}
probabilities = softmax_probabilities.eval(feed_dict=dict)
preds = predictions.eval(feed_dict=dict)
top_k_3 = sess.run(tf.nn.top_k(softmax_probabilities, k=3), feed_dict=dict)
top_k_5 = sess.run(tf.nn.top_k(softmax_probabilities, k=5), feed_dict=dict)
titles = ["Truth: %d, Pred: %d"%(external_labels[idx], preds[idx]) for idx in range(len(external_X))]
show_images(external_images_resized, titles, 3)
num_correct_predictions = np.count_nonzero(preds == external_labels)
num_examples = len(external_labels)
print("Number of examples: %d, Correct Predictions: %d, Accuracy = %.3f"%(num_examples, num_correct_predictions, num_correct_predictions/num_examples))
Question 7¶
Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.
NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.
Answer:
My model only predicts 2 out of the 13 new images correctly. This is a poor accuracy of 0.154. I will take a look at transforming training data images by rotating them various angles and changing perspective slightly. Hopefully that will give me better accuracy.
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.
print("Printing winning probabilities:")
for idx in range(len(external_labels)):
print("%.3f"%probabilities[idx][preds[idx]])
def get_top_n_info_label(values, indices):
assert len(values) == len(indices)
str = ""
for idx in range(len(indices)):
str += "%d:%.2f, "%(indices[idx], values[idx])
return str
titles = []
top_3_wins = 0
top_5_wins = 0
for idx in range(len(external_X)):
truth = external_y[idx]
top_k_info = get_top_n_info_label(top_k_3.values[idx], top_k_3.indices[idx])
if truth in top_k_3.indices:
top_3_wins += 1
if truth in top_k_5.indices:
top_5_wins += 1
titles.append("Truth: %d, TopK: %s"%(truth, top_k_info))
print("Accuracy using top K predictions:")
print("K = 3: %.3f"%(top_3_wins/num_examples))
print("K = 5: %.3f"%(top_5_wins/num_examples))
show_images(external_images_resized, titles, 1)
Question 8¶
Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)
tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.
Take this numpy array as an example:
# (5, 6) array
a = np.array([[ 0.24879643, 0.07032244, 0.12641572, 0.34763842, 0.07893497,
0.12789202],
[ 0.28086119, 0.27569815, 0.08594638, 0.0178669 , 0.18063401,
0.15899337],
[ 0.26076848, 0.23664738, 0.08020603, 0.07001922, 0.1134371 ,
0.23892179],
[ 0.11943333, 0.29198961, 0.02605103, 0.26234032, 0.1351348 ,
0.16505091],
[ 0.09561176, 0.34396535, 0.0643941 , 0.16240774, 0.24206137,
0.09155967]])Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:
TopKV2(values=array([[ 0.34763842, 0.24879643, 0.12789202],
[ 0.28086119, 0.27569815, 0.18063401],
[ 0.26076848, 0.23892179, 0.23664738],
[ 0.29198961, 0.26234032, 0.16505091],
[ 0.34396535, 0.24206137, 0.16240774]]), indices=array([[3, 0, 5],
[0, 1, 4],
[0, 5, 1],
[1, 3, 5],
[1, 4, 3]], dtype=int32))Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.
Answer:
Accuracy using top K predictions:
K = 3: accuracy = 0.692
K = 5: accuracy = 0.692
Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.